Image inpainting, also known as image completion, is a process of filling in missing data in a designated region of an image in a visually plausible manner. Conventionally, image inpainting has been performed by professional artists. With the rapid development of digital techniques, image inpainting has become a challenging research area in computer graphics and computer vision. Numerous interesting applications, from image restoration to object removal/replacement, have arisen. Many image rendering techniques, such as image filtering, sketch reality, and texture synthesis can be readily employed in image inpainting to recover the unknown regions with available information from their surroundings.
Image inpainting has attracted a lot of attention since it was introduced in the year 2000. Texture synthesis techniques are widely utilized in image inpainting to fill in regions that have textures that are homogenous with their surroundings. Some schemes synthesize at the pixel level while other approaches, such as image quilting, texture sampling, graph-cut techniques, and texture optimization, perform at a patch level. What these techniques have in common is that they work well for propagating a texture when little structural information exists in the unknown region, yet each has difficulties in reserving salient structure.
Partial differential equation (PDE)-based schemes fill in missing regions of an image by smoothly propagating image Laplacians in the isophote direction from the exterior. Exponential family distribution is used to model the image and the holes are filled in from the boundaries. Some schemes replace missing areas by interpolation guided by the estimated edge information. These pixel-based approaches produce good results for small flaws and thin structures. However, large-scale missing regions are still a challenging problem in such schemes.
Recently, patch-based approaches have been proposed for image inpainting by augmenting texture synthesis with certain automatic guidance. This preserves better edge sharpness and structure continuity. The recovery of the unknown region is gradually completed by a composition of circular known patches guided by a smooth approximation. A best-first algorithm imposes higher priorities on pixels lying in neighborhoods of edges before a restoration is performed according to the priority order at patch level. Additionally, the methods—pixel-based inpainting and patch-based synthesis—are integrated to suit different local characteristics. They are therefore capable of propagating both regular (linear or simple curvilinear) structures as well as 2-dimensional texture into the unknown regions.
Moreover, interactive guidance has been used to further improve inpainting capability by indicating the source region, depth information, and important salient structure. With high-level knowledge employed, these approaches can produce good results for certain challenging images. However, it is still difficult for such inpainting schemes to complete areas that feature complex and even semantic structures, especially in instances that have irregular structure or in which the majority or entire object is missing.